Image merge device and method

ABSTRACT

The image merge device includes a common area determination unit configured to determine a common area between a first image and a second image; a correlation calculation unit configured to calculate a correlation level indicating a degree of a gap between the first image near a boundary of the first image and the second image and the second image near the boundary when the first image and the second image are aligned using the common area; and a superimposed area determination unit configured to determine a superimposed area in which the first and second images are superimposed near the boundary based on the correlation level calculated by the correlation calculation unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of international applicationPCT/JP2007/ 001036, which was filed on Sep. 25, 2007, the entirecontents of which are incorporated herein by reference.

FIELD

The present invention relates to a method and a device for merging aplurality of images.

BACKGROUND

When an image is obtained by combining a plurality of images, anoverlapping or common area between a plurality of images to be combinedis set as an “overlapping area” (hereinafter an overlapping area isreferred to as a common area). The relative positions between theplurality of images are calculated, and the positions of the pluralityof images are aligned based on the relative positions. A composite imageis generated by merging the plurality of images after the alignment.However, since the plurality of images are obtained at different timingwith different quantity of light due to camera angle and so on, thereoccurs an unnatural break at the boundary of merged images (joint ofimages).

Therefore, a plurality of images are superimposed near the boundary(hereinafter referred to as a superimposed area) and merged in such away that the boundary looks natural. Accordingly, the resultantcomposite image includes an area generated from only one image and anarea generated by superposing a plurality of images.

When a moving object is shot by a camera and the moving object exists ina superimposed area in the composite image, there may be ghost images(two or more shifted images) of the moving object in the superimposedarea. Therefore, it is desired that the superimposed area is smaller.However, when there are large gaps between a plurality of images to bemerged, and the superimposed area is very small, the discontinuity atthe boundary due to the gaps between the plurality of images cannot besuppressed.

To solve the above-mentioned problem, a method in which the width of asuperimposed area may be selected by a user of a digital camera isproposed. This method is proposed, for example, by Japanese Laid-openPatent Publication No. 2000-299804.

However, the proposed method requires an operation by a camera user, andthe operation is complicated. Thus, the quality of an image depends onthe experience or technique of the camera user. In addition, since thewidth of the superimposed area is selected in advance, the compositeimage sometimes has a discontinuity at the boundary of a plurality ofimages.

SUMMARY

According to an aspect of an invention, the image merge device, whichmerges a first image and a second image, includes a common areadetermination unit configured to determine a common area between thefirst image and the second image; a correlation calculation unitconfigured to calculate a correlation level indicating a degree of a gapbetween the first image near a boundary of the first image and thesecond image and the second image near the boundary when the first imageand the second image are aligned using the common area; and asuperimposed area determination unit configured to determine asuperimposed area in which the first and second images are superimposednear the boundary based on the correlation level calculated by thecorrelation calculation unit.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and arenot restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates the image merge device according to the embodiments;

FIG. 2 is a configuration of the image merge device according to thefirst embodiment;

FIG. 3 is a flowchart of the process performed by the image merge deviceaccording to the first embodiment;

FIG. 4 is an explanatory view of a common area when two images areshifted in the 2-dimensional direction;

FIGS. 5A and 5B are explanatory views of an area for which a correlationlevel is calculated;

FIGS. 6A and 6B are explanatory views of a superimposed area;

FIG. 7 is an example of the superimposition width determinationfunction;

FIG. 8 is a configuration of the image merge device according to thesecond embodiment;

FIG. 9 is a flowchart of the process performed by the image merge deviceaccording to the second embodiment;

FIG. 10 is an explanatory view of a method in which the merge rate ischanged depending on the distance from the boundary;

FIG. 11A through 11C are explanatory views of the vicinity of theboundary when the gap between the images is large;

FIG. 12A through 12C are explanatory views of the vicinity of theboundary when the gap between the images is small;

FIGS. 13A and 13B are explanatory views of the case in which the mergerate is changed in a superimposition range for each area;

FIG. 14 is a configuration of the image merge device according to thethird embodiment;

FIG. 15 is a flowchart of the process performed by the image mergedevice according to the third embodiment;

FIG. 16 is an explanatory view of the area near the boundary where aconversion is performed;

FIG. 17 is an example of a gamma correction curve;

FIG. 18 is an example of a correspondence chart between the differenceof Y component and gamma correction curve; and

FIG. 19 is an explanatory view of a method in which a number of imagesare combined.

DESCRIPTION OF EMBODIMENTS

The embodiments of the present invention are described below withreference to the attached drawings. These embodiments are describedfully in detail so that they can be realized by those skilled in theart, but other embodiments are also available. For example, changes inconfiguration, logic, and electrical circuit to an embodiment may bemade within the scope of the subject of the present invention.Therefore, the following descriptions are not to be restrictivelyunderstood, but the scope of the subject of the invention is to bedefined by the scope of the claims of the patent and their equivalences.

Furthermore, the functions of the device described below may be realizedby software, hardware, firmware, and their optional combinations. Theexamples described below may have a single function or a combination offunctions for each unit, and other combinations of functions may also berealized within the scope of the subject of the present invention.

FIG. 1 illustrates an image merge device of the embodiments. Asillustrated in FIG. 1, an image merge device 10 according to theembodiments includes: a correlation calculation unit 11 for calculatingthe correlation level indicating the degree or level of the gap betweena plurality of images; a superimposed area determination unit 12 fordetermining a superimposed area in which the plurality of images aresuperimposed according to the correlation level; and a merge unit 13 (orimage generation unit) for merging the plurality of images bysuperimposing the images in the superimposed area.

When a plurality of aligned images are input, the correlationcalculation unit 11 calculates the correlation level indicating thedegree indicating the gap between the plurality of images near theboundary where the plurality of images are to be combined. To be morepractical, the correlation calculation unit 11 acquires a pixel near theboundary from each of the plurality of images, calculates the statisticsof the pixel values of each image, and calculates the correlation levelbased on the statistics. The higher the degree of similarity between theimages is, that is, the smaller the gap between the images is, thehigher the correlation level calculated by the correlation calculationunit 11 becomes.

The vicinity of the boundary of the joint refers to the area within acertain width in the plurality of images from the boundary. Therefore, ahigh correlation level indicates no large gap between the plurality ofimages near the boundary.

The statistics of pixel values used in calculating the correlation levelmay be at least one of the difference between the brightness, chroma orsaturation, color temperature, and pixel value. In addition, thestatistics may be an average, maximum, cumulative sum, standarddeviation or dispersion of them.

For example, when the correlation level is calculated based on thebrightness, a high correlation level indicates a small cap in brightnessbetween the plurality of images near the boundary.

The superimposed area determination unit 12 determines an area in whichthe images are superimposed near the boundary of the joint. The mergeunit 13 (or image generation unit) combines and merges the plurality ofimages by superimposing the plurality of images in the superimposedarea.

When a plurality of images are combined and edge areas of the images aresuperimposed at the boundary, the boundary can be blurred. However, ifthe superimposed area in which the images are superimposed is fixed, thesuperimposed area may be too large or small depending on the gap betweenthe images. For example, if the superimposed area is fixed when the gapbetween images are sufficiently small, the superimposed area is toolarge, and ghost images of an object may be generated in thesuperimposed area. On the other hand, if the superimposed area is fixedwhen the gap between the images is large, the superimposed area is toosmall and it may not be possible to solve the problem of discontinuityat the boundary.

The image merge device 10 according to the embodiments avoids the ghostimages in the superimposed area while reserving a necessary superimposedarea to suppress the gap between the images by determining asuperimposed area based on the correlation level. That is, when the gapbetween the images is sufficiently small, the superimposed areadetermination unit 12 avoids the ghost images in the superimposed areaby reducing the superimposed area. When the gap between the images islarge, the superimposed area determination unit 12 eliminates thediscontinuity occurring at the boundary of the joint by enlarging thesuperimposed area.

Since the superimposed area determination unit 12 determines thesuperimposed area based on the plurality of combined images, it is notnecessary for a camera user to set the superimposed area before shootingan image. Therefore, the quality of an image does not depend on theexperience of a camera user or the shooting technique. In addition,since the superimposed area is determined with respect to the obtainedimages, not determined in advance, the problem that the superimposedarea proves to be inappropriate after practically shooting an image doesnot occur.

In addition, the merge rate of a plurality of image may be changeddepending on the distances from the boundary. According to this method,since a plurality of images are smoothly merged in the superimposedarea, it is possible to blur the boundary more effectively.

In addition to the above-mentioned configuration, the image merge device10 may be provided with an image conversion unit for comparing thecorrelation level with a predetermined value, and converting at leastone of a plurality of images to be merged based on a result of thecomparison. In this case, the image conversion unit converts at leastone of the plurality of images so that the correlation level between theplurality of images is high. That is, the image conversion unit convertsan image so that the gap between a plurality of images become small.

According to this configuration, when the gap between the images islarge, the images is not only superimposed, but also converted so thatthe gap between the images become small in an area near the boundary,thereby further blurring the boundary of the joint.

The image conversion unit may convert the vicinity of the boundary ofthe image to be converted. If the entire images are converted when anumber of images are merged, the images may be largely different fromthe original images. According to the above configuration, the problemis overcome by limiting the range of the conversion to the vicinity ofthe boundary.

The image conversion unit may compare the correlation level between theplurality of converted image with a predetermined value, and repeat theconversion until a comparison result satisfies a predeterminedcondition. In this case, the image conversion unit may repeat theconversion until the correlation level between the plurality ofconverted images becomes smaller than a predetermined value. Thus, theboundary of the joint becomes furthermore blurred.

In addition, the image synthesis device 10 may be further provided witha shift amount calculation unit for calculating the amount of shift of aplurality of images, and a common area determination unit for aligningthe plurality of images based on the calculated amount of shift, anddetermining the area overlapping between the plurality of images as acommon area. According to this configuration, a plurality of image forwhich alignment and common area are not set in advance may be merged.

Furthermore, the image merge device 10 may be provided for an imageshooting device, such as a digital camera or an electronic camera, andso on. The image shooting device having the image merge device may beprovided for a mobile telephone terminal, a PDA (personal digitalassistant), a personal computer, etc.

A method including the processes performed by corresponding unitsprovided in the image merge device 10 also relates to the embodiments ofthe present invention, and the method may attain the above-mentionedobjectives.

In addition, a program used to direct a processor to perform controlsimilar to the functions of corresponding unit provided in the imagemerge device 10, a record medium recording the program, and a programproduct are also related to the embodiments of the present invention.The above-mentioned objects may also be attained by allowing a processorto read the program and control various interfaces etc. connectedthereto based on the program.

The following description is made by assuming that two images aremerged. The assumption is made for easy explanation, and two or moreimages may also be merged. Furthermore, the images are described asrectangular images, but may also be in any forms other than rectangles.

Described below is the image merge device according to the firstembodiment. An image merge device 100 according to the first embodimentdetermines a superimposed area by considering the gap between the imagesnear the boundary at which the images are combined, and merges twoimages by superimposing the two images in the superimposed area.

FIG. 2 is a configuration of the image merge device 100 according to thefirst embodiment. As illustrated in FIG. 2, the image merge device 100includes a shift amount calculation unit 110, a common areadetermination unit 120, a correlation calculation unit 130, asuperimposed area determination unit 140, a selector 150, a merge unit160, and an image generation unit 170.

The shift amount calculation unit 110 receives images P1 and P2 to becombined. The shift amount calculation unit 110 calculates an amount ofshift by matching corresponding pixels of the images P1 and P2. Thecommon area determination unit 120 aligns the images P1 and P2 based onthe calculated amount of shift, and determines the overlapping area ofthe two images as a common area. The correlation calculation unit 130calculates the correlation level indicating the degree of the gapbetween the images P1 and P2 near the boundary of the common area basedon of the pixels of the images P1 and P2. The superimposed areadetermination unit 140 determines a superimposed area as an area inwhich the images P1 and P2 are to be superimposed based on thecalculated correlation level. The selector 150 outputs the pixels in theinput images P1 and P2 to the merge unit 160 or the image generationunit 170. The merge unit 160 merges the pixels of the images P1 and P2,and outputs the resultant pixels to the image generation unit 170. Theimage generation unit 170 generates an image after the merge processing(hereinafter referred to as a composite image) from the pixels outputfrom the selector 150 and the merge unit 160, and outputs the image.

The flow of the process performed by the image merge device 100 isdescribed below with reference to FIG. 3.

First, the shift amount calculation unit 110 calculates the amount ofshift by matching corresponding pixels of the images P1 and P2 (stepS1). The common area determination unit 120 performs an alignment byconverting coordinates of the image P1 or P2 so that there is no shiftbetween the images according to the calculated amount of shift.Furthermore, the common area determination unit 120 determines theoverlapping area between the two aligned images as a common area (stepS2).

The correlation calculation unit 130 calculates the correlation levelnear the boundary of a common area (step S3). To be more practical, thecorrelation calculation unit 130 retrieves a pixel near the boundaryfrom each of the images P1 and P2 according to the coordinateinformation of the pixel, calculates the statistics about the pixels,and calculates the correlation level based on the statistics. There arevarious calculating methods about the correlation level. For example,there are calculating methods using the statistics of the brightness,the statistics of the chroma, the statistics of the color temperature,and/or the statistics of the difference between pixel values. When thestatistics of the brightness, the chroma and/or color temperature areused, for example, an average value, a standard deviation, a dispersion,etc. of them maybe used. When the statistics of the difference betweenthe pixel values is used in a calculating method, the statistics may bean average, a maximum, a cumulative sum, a standard deviation, adispersion, etc.

Then, the superimposed area determination unit 140 determines thesuperimposed area based on the calculated correlation level. To be morepractical, the superimposed area determination unit 140 determines asmall superimposed area when the correlation level calculated by thecorrelation calculation unit 130 is high, and determines a largesuperimposed area when the calculated correlation level is low (stepS4).

The selector 150 determines according to the determined superimposedarea and the coordinate information about the pixels of the images P1and P2 whether or not each pixel is in the superimposed area of themerged images (step S5). If it is determined that a pixel is within thesuperimposed area (YES in step S5), the selector 150 outputs the pixelwith its coordinate information to the merge unit 160. The merge unit160 merges the relevant pixel of the image P1 and the pixel of the imageP2 in the position corresponding to the relevant pixel of the image P1,and outputs the pixel obtained as a result of the merge process togetherwith the coordinate information about the merged image. The merge rateof the images P1 and P2 may be an arbitrary constant or a variable (stepS6). On the other hand, when it is determined that a pixel is not in thesuperimposed area, the selector 150 outputs the pixel as is togetherwith the coordinate information about the pixel to the image generationunit 170 (step S7). If a pixel is in the common area but not in thesuperimposed area, and there are pixels of the images P1 and P2, thenthe selector 150 outputs a pixel of one of the images and the coordinateinformation about the composite image.

If the above-mentioned processes are performed on the entire area of theimages P1 and P2 (YES in step S8), the image generation unit 170generates an composite image according to the pixels and the coordinateinformation output from the selector 150 and the merge unit 160 (stepS9).

The area near the boundary about which the correlation level iscalculated is described below with reference to FIGS. 4, 5A and 5B. FIG.4 illustrates the common area and the vicinity of the boundary when theimages P1 and P2 to be merged are shifted vertically and horizontally.The common area is an overlapping area between the images P1 and P2, andis represented by diagonal lines in FIG. 4. The common area is arectangle enclosed by a part of the periphery of the image P1 and a partof the periphery of the image P2. Hereinafter, in each side of thecommon area, the two sides configured by a part of the periphery of theimage P1 are referred to as the boundary of the image P1, and the twosides configured by a part of the periphery of the image P2 are referredto as the boundary of the image P2.

When two images are merged, the image P1 or the image P2 is used in thecommon area as described below with reference to FIGS. 5A and 5B. Asillustrated in FIGS. 5A and 5B, the boundary of the image used in thecommon area appears on the composite image, and the boundary of theimage not used in the common area is hidden behind the composite image.It is not necessary to calculate the correlation level about thevicinity of the hidden boundary. Therefore, the area for which thecorrelation calculation unit 130 calculates the correlation level isseveral pixels distant from the boundary of the image used in the commonarea. In FIGS. 5A and 5B, the area represented by the diagonal lines isan area for which the correlation level is calculated.

That is, when the image P1 is used in the common area, the boundary ofthe image P1 appears on the composite image as illustrated in FIG. 5A,and the area within a range inward and outward the common area from theboundary of the image P1 is selected as an area for which thecorrelation level is calculated. Similarly, when the image P2 is used inthe common area, the area within a range inward and outward the commonarea from the boundary of the image P2 is selected as an area for whichthe correlation level is calculated as illustrated in FIG. 5B. The widthof the area near the boundary may be within several percent through 20percent of the entire width of an image.

Next, the calculation of the correlation level is described below indetail. The correlation calculation unit 130 calculates the correlationlevel using the statistics of pixel values such as the statistics of thebrightness, the statistics of the chroma, the statistics of the colortemperature, the statistics of the difference between pixel values etc.

For example, the case in which an average of brightness is used tocalculate the correlation level is described below. In this example, theYCbCr color space or the YPbPr color space frequently used in TV etc.are used. In these color spaces, the brightness is represented by Ycomponent. It is only an example, and there are various color spacesdepending on the devices, and any color space may be applied in thesimilar method.

First, the correlation calculation unit 130 acquires pixels located nearthe boundary of the common area from each of the images P1 and P2, andcalculates an average of the Y component of the pixels from the imageP1, and an average of the Y component of the pixels from the image P2.Furthermore, the correlation calculation unit 130 calculates thedifference between the average values of the Y values. Then, thecorrelation calculation unit 130 may define the reciprocal of the“absolute value of the difference +1”, or the reciprocal of a valueobtained by dividing the “absolute value of the difference +1” by the“gray scale level (in many cases, the Y component is represented by 256levels from 0 to 255)” as the correlation level. As another example tocalculate the correlation level, the case in which the average of thelightness in the HSB (Hue saturation brightness) color space (HSV (Huesaturation value) color space) is used is described below. It is obviousthat this is an example only. In this case, the correlation calculationunit 130 acquires pixels located near the boundary of the common areafrom each of the images P1 and P2, and calculates the average value ofthe lightness of the pixels from the image P1 and the average value ofthe lightness of the pixels from the image P2. Furthermore, thecorrelation calculation unit 130 calculates the difference between thetwo obtained average values of the lightness. The correlation level maybe a reciprocal of the “absolute value of the difference +1”. Similarlywhen another statistics is used, the correlation calculation unit 130calculates the statistics for each of the pixels from the vicinity ofthe boundary of the image P1 and the vicinity of the boundary of theimage P2, and calculates the correlation level based on the statistics.In this case, the correlation calculation unit 130 calculates thecorrelation level so that the higher similarity of the vicinities of theboundaries of the images P1 and P2 are indicated by the statistics, thehigher correlation level is attained. That is, the smaller the gapbetween the images in the area near the boundaries, the higher thecorrelation level.

When the correlation level is calculated, the superimposed areadetermination unit 140 determines a superimposed area. With reference toFIGS. 6A and 6B, the superimposed area is described.

FIGS. 6A and 6B illustrate the superimposed areas when the images P1 andP2 are shifted from each other in vertical direction and horizontaldirection. A common area is an area in which the images P1 and P2overlap each other, and the common area is represented by the diagonallines in FIGS. 6A and 6B. The superimposed area is a part of the commonarea, and represented by the shaded area in FIGS. 6A and 6B. The shapeof the superimposed area is like a ‘hook” in this example. When theimage P1 is used in the common area, the boundary of the image P1appears in the composite image as illustrated in FIG. 6A, and the areain the common area within a certain distance from the boundary of theimage P1 is determined as the superimposed area. Similarly, when theimage P2 is used in the common area, the area in the common area withina certain distance from the boundary of the image P2 is determined asthe superimposed area as illustrated in FIG. 6B. The width of thesuperimposed area may be in the several percent to 20 percent of theentire width of the common area.

Next, the determination of the superimposed area by the superimposedarea determination unit 140 is described below. When the calculatedcorrelation level is high, the superimposed area determination unit 140determines a small superimposition range, and when the correlation levelis low, it determines a large superimposition range.

That is, when the correlation level is high, the gap between the imagesnear the boundary is small. Therefore, the boundary of the common areabetween the images P1 and P2 is not outstanding. Accordingly, anunnatural break hardly occurs at the boundary when the two images aremerged. Therefore, the superimposed area determination unit 140 providesa narrow superimposed area to avoid ghosts in the superimposed area. Onthe other hand, when the correlation level is low, the gap between theimages at the boundary is large. Accordingly, there occurs an unnaturalbreak at the boundary when the two images are merged. Therefore, thesuperimposed area determination unit 140 provides a broad superimposedarea, thereby solving the problem of the discontinuity near theboundary.

Thus, the image merge device 100 determines the superimposed areadepending on the gap near the boundary of the two images to be merged,thereby solving the problem of discontinuity near the boundary whileavoiding ghosts in the superimposed area.

Even when the correlation level is very high, if the superimposed area,that is, the superimposition width, is too small, there may be adiscontinuity at the boundary. In addition, a superimposition widthexceeding the common area cannot be set (refer to FIGS. 6A and 6B).Therefore, with the above-mentioned considerations taken into account,the superimposed area determination unit 140 may be assigned the maximumvalue and the minimum value of the superimposition width. For example,the maximum value of the superimposition width may be set based on thedetermination of the common area by the common area determination unit120.

Furthermore, the superimposed area determination unit 140 may use asuperimposition width determination function illustrated in FIG. 7. Inthis case, the superimposed area determination unit 140 may determinethe superimposition width within a range from the minimum value and themaximum value according to the function.

In one example, when the correlation level is calculated using thebrightness, that is, the Y component in the YCbCr color space or theYPbPr color space, the value which is obtained by dividing the absolutevalue of the difference between the average values of the Y component by255 is calculated. In this case, when the calculated value is about 10percent, it is considered that the gap between the images is large. Ifthe value is about 4 percent, it is considered that the gap between theimages is sufficiently small. Therefore, the superimposed areadetermination unit 140 may provide the maximum superimposition widthwhen the calculated value is higher than 10 percent. Then, thesuperimposed area determination unit 140 may gradually decrease thesuperimposition width as the calculated value decreases, and provide theminimum superimposition width when the calculated value is lower than 4percent.

In addition, for example, when the correlation level is calculated usinga color temperature, the absolute value of the difference in colortemperature is acquired in the process of calculating the correlationlevel. In this case, when the value is about 1500 kelvins, it may beconsidered that the gap between the images is large. Therefore, themaximum superimposition width is used when the calculated value is 1500kelvins or more. On the other hand, when the calculated value is about300 kelvins, it may be considered that the gap between the images issufficiently small. Therefore, the minimum superimposition width is usedwhen the calculated value is 300 kelvins or less.

The maximum value and the minimum value of the superimposition width maydepend on a policy to improve the quality of the composite image. Thatis, when the solution to remove the discontinuity is highly considered,it is desired that relatively large values are used for the maximumvalue and the minimum value. On the other hand, when the solution toremove the ghosts is highly considered, it is desired that relativelysmall values are used for the maximum value and the minimum value.

A plurality sets of the maximum value and the minimum value may beprepared in advance in the superimposed area determination unit 140. Inthis case, the user may select one set of the maximum value and theminimum value. For example, if avoiding the discontinuity is moreimportant than ghosts for the user, the user may select relatively largemaximum and minimum values. Thus, a user-requested superimpositionwidth, that is, superimposed area, may be determined.

Next, the image merge device according to the second embodiment isdescribed below. An image merge device 200 according to the secondembodiment has the function of changing the merge rate used insuperposing two images in the superimposition area.

FIG. 8 is the configuration of the image merge device 200 according tothe second embodiment. As illustrated in FIG. 8, the image merge device200 is further provided with a merge rate calculation unit 210 inaddition to the units included in the image merge device 100.

The merge rate calculation unit 210 calculates a merge rate of aplurality of images in a superimposed area for each pixel or area, andthe merge unit 160 merges the images based on the calculated merge rate.The operations of other units has already been described.

The flow of the process performed by the image merge device 200 isdescribed below with reference to FIG. 9. As illustrated in FIG. 9, theimage merge device 200 performs the process of calculating a merge ratein step S20 between steps S5 and S6. Other processes are similar tothose performed by the image merge device 100.

The calculation of the merge rate by the merge rate calculation unit 210is described below. First, the case in which a merge rate is changeddepending on the distance from the boundary is described below withreference to FIG. 10. For simple explanation, it is assumed thathorizontally shifted images P1 and P2 are combined, but the case inwhich they are shifted from each other in a two-dimensional direction isbasically the same. In FIG. 10, the image P2 is drawn smaller than theimage P2 for simple explanation. However, the two images are in the samesize.

FIG. 10 illustrates an example where images P1 and P2 are merged and theimage P2 is used in the common area. In this case, as illustrated inFIG. 10, the boundary of the image P2 appears on the composite image.FIG. 10 also illustrates a graph indicating the relationship between thedistance from the boundary of the image P2 and the merge rate. Thehorizontal axis of the graph in FIG. 10 indicates the distance from theboundary of the image P2, and the vertical axis indicates the mergerate. As illustrated in FIG. 10, at the boundary of the common area(boundary of the image P2), the merge rate of the image P1 is set as 1.0(100 percent), and the merge rate of the image P2 is set as 0. Then, themerge rate of the image P1 is reduced from 1.0 to 0, and the merge rateof the image P2 is increased from 0 to 1.0 (100 percent) in the regionstarting from the boundary of the common area to the end of thesuperimposed area. Thus, the images P1 and P2 are smoothly merged in themerge area.

Contrary to the case illustrated in FIG. 10, the merge rate of the imageP2 is 1.0 (100 percent), and the merge rate of the image P1 is 0 at theboundary of the common area (boundary of the image P1) when the image P1is used in the common area. Then, the merge rate of the image P2 isreduced from 1.0 to 0, and the merge rate of the image P1 is increasedfrom 0 to 1.0 as the distance from the boundary increases in the commonarea (not illustrated in the attached drawings).

The relationship between the size of the gap between images and thesuperimposition width is described below with reference to FIGS. 11A-11Cand 12A-12C. FIGS. 11A, 11B, and 11C illustrate the vicinity of theboundary of the images P1 and P2 when the gap between the images islarge. FIGS. 12A, 12B, and 12C illustrate the vicinity of the boundarywhen the gap between the images P1 and P2 is small.

As illustrated in FIG. 11A, when the gap between the images P1 and P2 islarge, an outstandingly discontinuity occurs at the boundary when theimages P1 and P2 are merged. In this case, unless the superimpositionwidth is sufficiently large, the discontinuity near the boundary isoutstanding although the images P1 and P2 are superimposed whilechanging the merge rate as illustrated in FIG. 11B. When thesuperimposition width is sufficiently large, the discontinuity near theboundary is hardly outstanding as a result of superposing the images P1and P2 while changing the merge rate as illustrated in FIG. 11C.

On the other hand, as illustrated in FIG. 12A, when the gap between theimages P1 and P2 is small, the discontinuity occurs at the boundary whenthe images P1 and P2 are merged, but it is not outstanding. In thiscase, even though the superimposition width is small, the discontinuitynear the boundary is hardly outstanding as a result of superimposing theimages P1 and P2 while changing the merge rate as illustrated in FIG.12B. When the superimposition width is large, the discontinuity near theboundary is furthermore hardly outstanding as a result of superimposingthe images P1 and P2 while changing the merge rate as illustrated inFIG. 12C. However, there is no conspicuous different between FIGS. 12Band 12C. In this case, avoiding the ghosts is highly considered morethan the suppressing the discontinuity. Therefore, it is sometimespreferable to select a small superimposition width in place of a largesuperimposition width.

Next, a method in which the merge rate in the superimposition range ischanged for each area is described with reference to FIGS. 13A and 13B.FIGS. 13A and 13B illustrate the exemplary cases in which thesuperimposition range is divided into two areas at the center of thesuperimposition area. The horizontal axes in the graph in FIGS. 13A and13B indicate the distances from the boundaries of the common area, andthe vertical axes indicate the merge rates. When the image P2 is used inthe common area, the boundary of the image P2 appears on the compositeimage. In this case, as illustrated in FIG. 13A, the merge rates of theimages P1 and P2 are respectively set as 1.0 (100 percent) and 0 fromthe boundary of the common area (boundary of the image P2) to the pointimmediately before the center of the superimposition area, and the mergerates of the images P1 and P2 are respectively set as 0.5 (50 percent)and 0.5 (50 percent) from the center of the superimposition area to thepoint immediately before the end of the superimposition area. Finally,the merge rates of the images P1 and P2 are respectively set as 0 and1.0 (100 percent) at the end of the superimposition area.

On the other hand, when the image P1 is used in the common area, theboundary of the image P1 appears on the composite image. In this case,as illustrated in FIG. 13B, contrary to the case illustrated in FIG.13A, the merge rate of the image P2 stepwise changes from 1.0 (100percent) to 0 through 0.5 (50 percent) from the boundary of the commonarea (boundary of the image P2) to the end of the superimposition width,and the merge rate of the image P1 stepwise changes from 0 to 1.0through 0.5.

Thus, the images P1 and P2 may be merged smoothly in the merge area bythe merge unit 160 superimposing the images P1 and P2 while stepwisechanging the merge rate for each area. In this case, the process issimpler than in the case in which the merge rate is changed for eachpixel as described above, thereby reducing the load on the processingdevice.

The image merge device according to the third embodiment is described.An image merge device 300 according to the third embodiment has thefunction of converting as necessary both or one of the images P1 and P2before merging the images in addition to the functions of the imagemerge device 100 according to the first embodiment.

FIG. 14 is a configuration of the image merge device 300 according tothe third embodiment. As illustrated in FIG. 14, the image merge device300 is provided with an image conversion unit 310 in addition to theunits of the image merge device 100.

The image conversion unit 310 converts both or one of the images toreduce the gap between the images when the correlation level is small,that is, when the gap between the images is large. Then, the correlationcalculation unit 130 calculates the correlation level between theimages.

The flow of the process performed by the image merge device 300 isdescribed below with reference to FIG. 15. As illustrated in FIG. 15,the image merge device 300 performs the processes in steps S30 throughS32 between steps S3 and S4. Described below are steps S30 through S32.

In step S3, when the correlation calculation unit 130 calculates thecorrelation level, the image conversion unit 310 compares thecorrelation level with a predetermined value (step S30). When thecorrelation level is lower than the predetermined value, that is, whenthe gap between the images is large (NO in step S30), the imageconversion unit 310 converts both or one of the plurality of images toreduce the gap between the images (step S31).

When the correlation level is higher than the predetermined value (YESin step S30), the image conversion unit 310 does not convert the imageand the correlation level is fed to the superimposed area determinationunit 140, then the processes in and after step S4 are performed.

When the image conversion is performed in step S31, the correlationcalculation unit 130 calculates the correlation level of the convertedimages (step S32). The calculated correlation level is output to thesuperimposed area determination unit 140, and the processes in and afterstep S4 are performed.

After step S32, control may be returned to step S30. In this case, theimage conversion unit 310 repeats the image conversion process until thecorrelation level becomes smaller than the predetermined value (notillustrated in the attached drawings). Other processes are similar tothose performed by the image merge device 100.

Described below is the image conversion by the image conversion unit310. The image conversion unit 310 determines that the gap between theimages is large when the correlation level is smaller than apredetermined value, and converts at least one of the images so that thegap between the images is reduced. The conversion process may beperformed on the entire image, or on the area near the boundary. Theconversion process may be performed on both or one of the two images tobe merged. When the vicinity of the boundary is converted, one side orboth sides of the boundary may be converted.

The area to be converted when a conversion is made to the vicinity ofthe boundary is described with reference to FIG. 16. For simpleexplanation, it is assumed that the images P1 and P2 are shifted fromeach other in the horizontal direction. The similar method may beapplied to the case where the images are shifted in the two-dimensionaldirection. In FIG. 16, the image P2 is drawn smaller than the image P1only for convenience in explanation, but the two images may be the samesize.

FIG. 16 illustrates a case where the images P1 and P2 merged and theimage P2 is used in the common area. FIG. 16 also illustrates a graphindicating the relationship between the distance from the boundary ofthe image P2 and the merge rate. The horizontal axis of the graph inFIG. 16 indicates the distance from the boundary of the image P2, andthe vertical axis indicates the Y component. As illustrated in FIG. 16,the area in which images are converted is within a predetermineddistance from the boundary. The conversion area may be the same as thearea for which the correlation level is calculated, but the conversionarea may be larger than the area. In addition, the conversion area maybe larger than the superimposed area. Depending on the size of theimage, the conversion area may be several percent through several tensof percent of the width of the entire image.

Described below is an example of the case in which an image is convertedbased on the brightness. In this example, the YCbCr color space or theYPbPr color space is used as an example. In these color spaces, thebrightness is expressed by the Y component.

Assuming that the an average value of the Y component of the image P1near the boundary is higher than the average value of the Y component ofthe image P2 (that is, the image P1 is brighter than image P2), theimage conversion unit 310 reduces the brightness of the image P1 in thevicinity of the boundary, and on the other hand, increases thebrightness of the image P2 in the vicinity of the boundary. As a result,as illustrated in FIG. 15, the brightness is gradually reduced withinthe conversion area from the image P1 to the image P2.

Described below is the image conversion by the image conversion unit310. When images are converted according to Y component, the imageconversion unit 310 may be provided with a plurality of gamma correctioncurves illustrated in FIG. 17 and a correspondence table of thedifference of the Y component and the gamma correction curvesillustrated in FIG. 18. The image conversion unit 310 calculates thedifference between the average value of the Y component of the image P1and the average value of the Y component of the image P2 when thecorrelation level calculated by the correlation calculation unit 130 islower than a predetermined value. Then the image conversion unit 310selects the gamma correction curve corresponding to the difference fromthe plurality of gamma correction curves illustrated in FIG. 17 based onthe correspondence table illustrated in FIG. 18. When there is no curvecorresponding to the difference, the image conversion unit 310 maygenerate a curve corresponding to the difference with reference to FIGS.17 and 18. Then, the image conversion unit 310 converts both or one ofthe images P1 and P2 according to the selected gamma correction curve.Afterward, the image conversion unit 310 outputs again the image P1and/or the image P2 after the image conversion to the correlationcalculation unit 130. The correlation calculation unit 130 calculatesthe correlation level on the converted images P1 and P2, and thesuperimposed area determination unit 140 determines the superimpositionwidth based on the correlation level.

The image conversion unit 310 may repeatedly perform the imageconversion until the correlation level between the images exceeds apredetermined value. In this case, the correlation calculation unit 130calculates the correlation level on the images P1 and P2, and returnsthe correlation level to the image conversion unit 310. Then the imageconversion unit 310 compares the correlation level with thepredetermined value to decide whether further conversion is necessary.

FIGS. 11A through 11C and 12A through 12C illustrate the vicinity of theboundary when the images P1 and P2 are converted so that the gap betweenthe images is reduced. For example, FIG. 11A illustrates the vicinity ofthe boundary in the case in which the image P2 is rather darker than theimage P1. In this case, an outstanding discontinuity occurs at theboundary. FIG. 11B illustrates an example where the image P2 iscorrected to be slightly brighter, and the image P1 is corrected to beslightly darker. With the effect of the small superimposition width, thediscontinuity that has occurred at the boundary still remains. FIG. 11Cillustrates a result of correcting the image P2 to be further brighter,and correcting the image P1 to be further darker. According to thisprocess, with the effect of the expanded superimposition width, thediscontinuity that has occurred at the boundary is hardly recognized.

FIGS. 12A through 12C illustrate a result of performing a similarconversion as in FIG. 11A through 11C when the gap between the images ismuch smaller than the case in FIG. 11A through 11C. In this case, asillustrated in FIG. 12B, since the discontinuity at the boundary becomesinconspicuous by one conversion process, the second image conversionprocess as illustrated in FIG. 11C may be omitted.

Next, the case in which a number of images are merged is described withreference to FIG. 19. FIG. 19 illustrates the superimposed areas whenfour images are merged, and the brightness of the right images comparedwith the respective left images. In this figure, the superimpositionwidths have nearly equal widths for each boundary, but it is obviousthat the superimposition width may be appropriately changed depending onthe correlation level of the images.

For example, when a plurality of images are continuously shot, thebrightness of the entire image may be different for each image due to achange of the position of a light source for the subject if the imagesare obtained indoors, and due to a change of the shape of cloud and soon if the images are obtained outdoors. In FIG. 19, the second image isbrighter than the first image, and the third image is further brighterthan the second image. In this case, if it is assumed that the imageconversion unit 310 converts the entire image so that the gap betweenthe images is reduced, then the conversion to brighten the imagescontinues. As a result, as the second image, the third image, and thesubsequent images in the rightward direction, the images becomegradually whitened.

However, as illustrated in FIG. 19, if the area in which a conversion ismade is limited to the vicinity of the boundary, then the discontinuitynear the boundary may be inconspicuous while avoiding theabove-mentioned problem.

The image merge devices 100, 200, and 300 according to the embodimentsmay be realized in various configurations. Described below are theimplementations of the image merge devices 100, 200, and 300.

The image merge devices 100, 200, and 300 may be realized by using acomputer. The computer (not illustrated in the attached drawings)includes at least a CPU (central processing unit processor), a memory,and an input/output interface. These components are interconnected via abus. The input/output interface may be, for example, a LCD panel, atouch-screen panel, various buttons, dials, etc.

To realize the image merge devices 100, 200, and 300 by using acomputer, a program used to direct a processor to perform a methodillustrated by the above flowcharts is stored in the memory. Then, theprocessor executes the program using the memory, thereby realizing theimage merge devices 100, 200, and 300.

Furthermore, the image merge device may be implemented on variousdevices. For example, the image merge device may be implemented on animage shooting device such as a still camera, video camera, and so on.

For example, the case in which the image merge device is implemented onan image shooting device is described below. An image shooting device isprovided with a lens, an image pickup device, a controller(non-general-purpose processor), a display device, a memory, aninput/output interface, and an auxiliary storage device in many cases. Alens forms an image of a subject on the image pickup device. The imagepickup device converts the formed image into an electronic signal andoutput the signal to the controller. The controller controls the memory,the storage device, the interface, and the display device. The memorystores an image merge program to perform an image merge methodillustrated by the above flowcharts. The controller reads the programfrom the memory, and executes the program, thereby realizing the imageshooting device implemented with the image merge device.

A computer chip (microcontroller) for enabling various interfaces toperform an image merge method illustrated by the above flowcharts may beprepared. In this case, the computer chip controls each equipment unitof the image shooting device, thereby realizing the image shootingdevice with the image merge device.

In addition, the image shooting device with the image merge device maybe realized by firmware to direct a processor to perform the image mergemethod. In this case, the firmware may be embedded in a memory of theimage shooting device.

In addition, the image shooting device having the image merge devices100, 200, and 300 may be implemented in a PDA (personal digitalassistant), a personal computer, etc.

Described below is loading a program. A program used to direct acomputer, an image shooting device, and a mobile telephone terminal, aPDA, etc. including an image shooting device to realize the function ofthe image merge device described according to the above-mentionedembodiments may be acquired in various methods.

For example, the program is stored in an external storage deviceconnected to the device (including a processor) for realizing the imagemerge device so that the program is loaded to memory as necessary.

A computer-readable record medium may store the program in advance, andthe program may be read by a device for realizing the image merge devicefrom the record medium. The memory of the computer or an externalstorage device may temporarily store the program, and the stored programmay be executed by the CPU of the computer.

The program may be stored by a storage device of a program server, and adevice for realizing the image merge device may download the programthrough an input interface and a communication circuit. In this case,for example, the program server converts program data representing theabove-mentioned program into a program data signal, and the convertedprogram data signal is modulated using a modulator, thereby acquiring atransmission signal, and the transmission signal is output to thecommunication circuit. A device for receiving the program demodulatesthe transmission signal by using a demodulator, thereby acquiring aprogram data signal, and the program data signal is converted to theprogram data.

When the communication circuit (transmission medium) for connecting thecomputers is a digital circuit, the program data signal may betransmitted. The program data signal may be relayed by a computer of acommunication service provider between a server computer fortransmitting a program and a client computer for downloading theprogram.

The embodiments described above may be arbitrarily combined. Otherchanges that can be made based on the descriptions and the drawings bythose skilled in the art correspond to the subject and the equivalentsof the invention.

According to the embodiments of the invention, when a plurality ofimages are combined and merged, the correlation level indicating thelevel (degree) of the gap between the plurality of images in an areanear the boundary of the images is calculated, and an area in which theplurality of images are superimposed is determined so that the boundaryof the merged images is not conspicuous based on the correlation level.Thus, the discontinuity at the boundary of the images is suppressed andthe ghosts are removed.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the inventionand the concepts contributed by the inventor to furthering the art, andare to be construed as being without limitation to such specificallyrecited examples and conditions, nor does the organization of suchexamples in the specification relate to a showing of the superiority andinferiority of the invention. Although the embodiment(s) of the presentinventions has (have) been described in detail, it should be understoodthat the various changes, substitutions, and alterations could be madehereto without departing from the spirit and scope of the invention.

1. An image merge device which merges a first image and a second image,comprising: a common area determination unit configured to determine acommon area between the first image and the second image; a correlationcalculation unit configured to calculate a correlation level indicatinga degree of a gap between the first image near a boundary of the firstimage and the second image and the second image near the boundary whenthe first image and the second image are aligned using the common area;and a superimposed area determination unit configured to determine asuperimposed area in which the first and second images are superimposednear the boundary based on the correlation level calculated by thecorrelation calculation unit.
 2. The image merge device according toclaim 1, wherein the correlation calculation unit calculates thecorrelation level for area within a predetermined width from theboundary when the first and second images are aligned using the commonarea.
 3. The image merge device according to claim 1, wherein thecorrelation calculation unit acquires pixels in the vicinity of theboundary from each of the first image and the second image, andcalculates the correlation level based on statistics of the pixels. 4.The image merge device according to claim 3, wherein the correlationcalculation unit calculates the correlation level based on at least oneof statistics of brightness, chroma, a color temperature, and adifference in pixel value.
 5. The image merge device according to claim3, wherein the statistics is any of an average value, a maximum value, acumulative sum, a standard deviation, and a dispersion.
 6. The imagemerge device according to claim 1, wherein the superimposed areadetermination unit determines a size of the superimposed area based onthe correlation level.
 7. The image merge device according to claim 1,wherein the superimposed area determination unit broadens thesuperimposed area when the correlation level is low.
 8. The image mergedevice according to claim 1, wherein the superimposed area determinationunit narrows the superimposed area when the correlation level is high.9. The image merge device according to claim 1, wherein a maximum valueand a minimum value of the superimposed area is assigned to thesuperimposed area determination unit.
 10. The image merge deviceaccording to claim 9, wherein the maximum value and the minimum value isarbitrarily set.
 11. The image merge device according to claim 1,wherein a merge rate of the first and second images in the superimposedarea changes depending on a distance from the boundary.
 12. The imagemerge device according to claim 1, further comprising an imageconversion unit configured to compare the correlation level with apredetermined value, and convert at least one of the first and secondimages based on a result of the comparison.
 13. The image merge deviceaccording to claim 12, wherein the image conversion unit converts atleast one of the first and second images so that the correlation levelafter the conversion become high.
 14. The image merge device accordingto claim 12, wherein the image conversion unit converts at least one ofthe first and second images so that a gap between images after theconversion become small.
 15. The image merge device according to claim12, wherein the image conversion unit converts the image in the vicinityof the boundary.
 16. The image merge device according to claim 12,wherein the image conversion unit repeats the conversion process untilthe result of the comparison satisfies a predetermined condition. 17.The image merge device according to claim 1, further comprising a shiftamount calculation unit configure to calculate an amount of shift of thefirst and second images, wherein the common area determination unitaligns the first and second images based on the amount of shift, anddetermines an area overlapping between the first and second images asthe common area, the boundary is an edge of the common area.
 18. Arecord medium storing an image merge program for enabling a computer toperform a image merge method, the image merge method comprising:determining a common area between the first image and the second image;calculating a correlation level indicating a degree of a gap between thefirst image near a boundary of the first image and the second image andthe second image near the boundary when the first image and the secondimage are aligned using the common area; and determining a superimposedarea in which the first and second images are superimposed near theboundary based on the correlation level.
 19. An image merge method formerging a first image and a second image, comprising: determining acommon area between the first image and the second image; calculating acorrelation level indicating a degree of a gap between the first imagenear a boundary of the first image and the second image and the secondimage near the boundary when the first image and the second image arealigned using the common area; and determining a superimposed area inwhich the first and second images are superimposed near the boundarybased on the correlation level.